Installation¶
Just pip install:
pip install omegaconf
If you want to try this notebook after checking out the repository be sure to run
python setup.py develop at the repository root before running this code.
xxxxxxxxxxfrom omegaconf import OmegaConfconf = OmegaConf.create()print(conf)From a dictionary¶
xxxxxxxxxxconf = OmegaConf.create(dict(k='v',list=[1,dict(a='1',b='2')]))print(OmegaConf.to_yaml(conf))From a list¶
xxxxxxxxxxconf = OmegaConf.create([1, dict(a=10, b=dict(a=10))])print(OmegaConf.to_yaml(conf))From a yaml file¶
xxxxxxxxxxconf = OmegaConf.load('../source/example.yaml')print(OmegaConf.to_yaml(conf))From a yaml string¶
xxxxxxxxxxyaml = """a: bb: clist:- item1- item2"""conf = OmegaConf.create(yaml)print(OmegaConf.to_yaml(conf))From a dot-list¶
xxxxxxxxxxdot_list = ["a.aa.aaa=1", "a.aa.bbb=2", "a.bb.aaa=3", "a.bb.bbb=4"]conf = OmegaConf.from_dotlist(dot_list)print(OmegaConf.to_yaml(conf))From command line arguments¶
To parse the content of sys.arg:
xxxxxxxxxx# Simulating command line argumentsimport syssys.argv = ['your-program.py', 'server.port=82', 'log.file=log2.txt']conf = OmegaConf.from_cli()print(OmegaConf.to_yaml(conf))Access and manipulation¶
Input yaml file:
xxxxxxxxxxconf = OmegaConf.load('../source/example.yaml')print(OmegaConf.to_yaml(conf))Object style access:¶
xxxxxxxxxxconf.server.portdictionary style access¶
xxxxxxxxxxconf['log']['rotation']items in list¶
xxxxxxxxxxconf.users[0]Changing existing keys¶
xxxxxxxxxxconf.server.port = 81Adding new keys¶
xxxxxxxxxxconf.server.hostname = "localhost"Adding a new dictionary¶
xxxxxxxxxxconf.database = {'hostname': 'database01', 'port': 3306}providing default values¶
xxxxxxxxxxconf.missing_key or 'a default value'conf.get('missing_key', 'a default value')Accessing mandatory values¶
Accessing fields with the value ??? will cause a MissingMandatoryValue exception. Use this to indicate that the value must be set before accessing.
xxxxxxxxxximport pytestfrom omegaconf import MissingMandatoryValuewith pytest.raises(MissingMandatoryValue): conf.log.fileInterpolations are evaluated lazily on field access.
Note below that when printed the interpolations are not resolved.
They get resolved once you access them.
xxxxxxxxxxconf = OmegaConf.load('../source/config_interpolation.yaml')print(OmegaConf.to_yaml(conf))xxxxxxxxxx# Primitive interpolation types are inherited from the referenced valueprint("conf.client.server_port: ", conf.client.server_port, type(conf.client.server_port).__name__)# Composite interpolation types are always stringprint("conf.client.url: ", conf.client.url, type(conf.client.url).__name__)to_yaml() will resolve interpolations if resolve=True is passed
xxxxxxxxxxprint(OmegaConf.to_yaml(conf, resolve=True))Interpolations may be nested, enabling more advanced behavior like dynamically selecting a sub-config:
xxxxxxxxxxcfg = OmegaConf.create("""plans: A: plan A B: plan Bselected_plan: Aplan: ${plans.${selected_plan}}""")print(f"Default: cfg.plan = {cfg.plan}")cfg.selected_plan = "B"print(f"After selecting plan B: cfg.plan = {cfg.plan}")xxxxxxxxxxfrom textwrap import dedentcfg = OmegaConf.create( dedent( """\ plans: A: plan A B: plan B selected_plan: A plan: ${plans.${selected_plan}} """ ))print(f"Default: cfg.plan = {cfg.plan}")cfg.selected_plan = "B"print(f"After selecting plan B: cfg.plan = {cfg.plan}")Environment variable interpolation¶
Environment variable interpolation is also supported.
xxxxxxxxxx# Let's set up the environment first (only needed for this demonstration)import osos.environ['USER'] = 'omry'Here is an example config file interpolates with the USER environment variable:
xxxxxxxxxxconf = OmegaConf.load('../source/env_interpolation.yaml')print(OmegaConf.to_yaml(conf))xxxxxxxxxxconf = OmegaConf.load('../source/env_interpolation.yaml')print(OmegaConf.to_yaml(conf, resolve=True))You can specify a default value to use in case the environment variable is not defined. The following example sets abc123 as the the default value when DB_PASSWORD is not defined.
xxxxxxxxxxos.environ.pop('DB_PASSWORD', None) # ensure env variable does not existcfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}'}})print(repr(cfg.database.password))OmegaConf.clear_cache(cfg) # clear resolver cacheos.environ["DB_PASSWORD"] = 'secret'print(repr(cfg.database.password))xxxxxxxxxxos.environ.pop('DB_PASSWORD', None) # ensure env variable does not existcfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}'}})print(repr(cfg.database.password))Environment variables are parsed when they are recognized as valid quantities that may be evaluated (e.g., int, float, dict, list):
xxxxxxxxxxcfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}', 'user': 'someuser', 'port': '${env:DB_PORT,3306}', 'nodes': '${env:DB_NODES,[]}'}})os.environ["DB_PORT"] = '3308'print(repr(cfg.database.port)) # converted to intos.environ["DB_NODES"] = '[host1, host2, host3]'print(repr(cfg.database.nodes)) # converted to listos.environ["DB_PASSWORD"] = 'a%#@~{}$*&^?/<'print(repr(cfg.database.password)) # kept as a stringxxxxxxxxxxcfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}', 'user': 'someuser', 'port': '${env:DB_PORT,3306}', 'nodes': '${env:DB_NODES,[]}'}})os.environ["DB_PORT"] = '3308' # integeros.environ["DB_NODES"] = '[host1, host2, host3]' # listos.environ["DB_PASSWORD"] = 'a%#@~{}$*&^?/<' # stringprint(repr(cfg.database.port))print(repr(cfg.database.nodes))print(repr(cfg.database.password))Custom interpolations¶
You can add additional interpolation types using custom resolvers. The example below creates a resolver that adds 10 to the given value.
xxxxxxxxxxOmegaConf.new_register_resolver("plus_10", lambda x: x + 10)conf = OmegaConf.create({'key': '${plus_10:990}'})conf.keyYou can take advantage of nested interpolations to perform custom operations over variables:
xxxxxxxxxxOmegaConf.new_register_resolver("plus", lambda x, y: x + y)conf = OmegaConf.create({"a": 1, "b": 2, "a_plus_b": "${plus:${a},${b}}"})conf.a_plus_bBy default a custom resolver’s output is cached, so that when it is called with the same inputs we always return the same value. This behavior may be disabled by setting use_cache=False:
xxxxxxxxxximport randomrandom.seed(1234)OmegaConf.new_register_resolver("cached", random.random)OmegaConf.new_register_resolver("uncached", random.random, use_cache=False)cfg = OmegaConf.create({"cached": "${cached:}", "uncached": "${uncached:}"})print("With cache: ")print(cfg.cached)print(cfg.cached) # same as aboveprint("Without cache: ")print(cfg.uncached)print(cfg.uncached) # *not* the same as aboveWith cache: 0.9664535356921388 0.9664535356921388 Without cache: 0.4407325991753527 0.007491470058587191
xxxxxxxxxximport randomrandom.seed(1234)OmegaConf.new_register_resolver("randint", lambda a, b: random.randint(a, b))c = OmegaConf.create({"x": "${randint:0, 1000}"})print("With cache:")print(f"c.x = {repr(c.x)}")print(f"c.x = {repr(c.x)}") # same as above thanks to the cacherandom.seed(1234)OmegaConf.new_register_resolver("randint_nocache", lambda a, b: random.randint(a, b), use_cache=False) # <== disable cache!c = OmegaConf.create({"x": "${randint_nocache:0, 1000}"})print("Without cache:")print(f"c.x = {repr(c.x)}")print(f"c.x = {repr(c.x)}") # not the same anymore since the cache is disabledWith cache: c.x = 989 c.x = 989 Without cache: c.x = 989 c.x = 796
Merging configurations¶
Merging configurations enables the creation of reusable configuration files for each logical component instead of a single config file for each variation of your task.
Machine learning experiment example:
conf = OmegaConf.merge(base_cfg, model_cfg, optimizer_cfg, dataset_cfg)
Web server configuration example:
conf = OmegaConf.merge(server_cfg, plugin1_cfg, site1_cfg, site2_cfg)
The following example creates two configs from files, and one from the cli. It then combines them into a single object. Note how the port changes to 82, and how the users lists are combined.
xxxxxxxxxxbase_conf = OmegaConf.load('../source/example2.yaml')print(OmegaConf.to_yaml(base_conf))xxxxxxxxxxsecond_conf = OmegaConf.load('../source/example3.yaml')print(OmegaConf.to_yaml(second_conf))xxxxxxxxxxfrom omegaconf import OmegaConfimport sys# Merge configs:conf = OmegaConf.merge(base_conf, second_conf)# Simulate command line argumentssys.argv = ['program.py', 'server.port=82']# Merge with cli argumentsconf.merge_with_cli()print(OmegaConf.to_yaml(conf))